Electronically Capturing Handwritten Data

ABSTRACT

A method and apparatus are provided for electronically capturing handwritten data using a computing device having a handwriting capture interface. The method includes detecting a position associated with an actuation of the handwriting capture interface to define a handwritten input, and defining a bounding box encompassing at least a portion of individual strokes of the handwritten input. In addition, the method includes storing the handwritten input and the bounding box.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.11/426,860, filed Jun. 27, 2006, which is a divisional of U.S.application Ser. No. 09/540,469, filed Mar. 31, 2000, now U.S. Pat. No.7,091,959, which claims the benefit of U.S. Provisional Application No.60/127,196, filed Mar. 31, 1999, wherein the contents of the abovelisted applications and patents are hereby incorporated by reference.

FIELD

The present invention relates generally to data processing with penenabled computing devices and, more particularly, to a system, computerprogram product, computing device, and associated methods forelectronically identifying preprinted forms used in conjunction with penenabled computing devices and accessing and manipulating informationassociated therewith.

BACKGROUND

Pen-enabled computing is a relatively recent development wherein a userinterfaces with a computing system by way of a writing stylus instead ofa mouse or a keyboard. In some instances, a pen-enabled computing systemprovides an electronic writing tablet having an overlaid writingsurface. Such a writing surface may comprise, for example, a blank sheetof paper or a preprinted form. The writing stylus, in turn, may comprisea device capable of inputting data (a “handwriting input”) into thepen-enabled computing system while providing both a visible, or “writtenink,” copy of the data on the writing surface and an “electronic” copyof the data within the pen-enabled computing system. Such a writingstylus may comprise, for example, a conventional pen, a conventionalpencil, a radio transmitter, a magnetic or electric field device, anultrasound transceiver, or combinations thereof.

Once the electronic copy of the data is entered into the pen-enabledcomputing system, the data may be stored as an “electronic ink” copy,wherein the handwriting input is captured as written, typically as asequence of (x, y) coordinate pairs, a bit map or the like, or as a“text” copy, wherein the handwriting input is captured, recognized, andtranslated into the corresponding text, typically by automatic onlinehandwriting recognition or the like. In some instances, the pen-enabledcomputing system may be capable of producing both an electronic ink anda text copy of the handwriting input. Since the writing stylus isgenerally capable of providing a written ink copy of the handwritinginput on the writing surface, the user is automatically provided with ahard copy, or visual feed back, of the entered data. It is understood,however, that a written ink copy of the handwriting input may not beprovided in some instances, wherein the handwriting input is then justcaptured by the electronic writing tablet and stored in the pen-enabledcomputing system as an electronic copy. Examples of pen enabled computerdevices include the CrossPad™ portable digital notepad by the A. T.Cross Company and the Palm Connected Organizer™ personal data assistant(PDA) by Palm, Inc.

One example of the capabilities of pen enabled computing devices isprovided by U.S. Pat. No. 5,629,499 to Flickinger et al., the contentsof which are incorporated by reference herein. Flickinger describes anelectronic board used to capture and store electronic ink data.According to Flickinger, a page having a writing surface is secured tothe electronic board with clip. Pen input data is then entered bywriting on the writing surface with a pen, wherein the writing iselectronically captured by the board to create an electronic ink copythereof. However, the electronic board is described by Flickinger asonly being capable of recording pen input data for subsequent transferof the pen input data to a processing device. Thus, there exists a needfor a pen enabled computing device having the innate capability toprocess or otherwise manipulate recorded pen input data without havingto transfer or otherwise interface with a separate or externalprocessing device.

The Flickinger electronic board is also described as being usable with ablank piece of paper or a preprinted form, wherein a user-initiatedoperation indicates to the board that a page has been positionedthereon. A subsequent operation is then required to identify the page tothe board. In one embodiment, the user must activate a switch outsidethe writing area to identify the page. However, having a separate switchfor indicating each type of form supported by the device limits thecapabilities and form type capacity thereof while disadvantageouslyadding components to the device. Further, a toggle switch requiringmultiple actuations in order to scroll through the list of form typessupported by the device may be inconvenient and may also limit thecapacity of the device. Thus, there exists a further need for a penenabled computing device having the capability to detect the presence ofa page engaged with the device and to determine the type of form with aminimum of separate identification devices and required user-initiatedoperations.

Another embodiment of Flickinger describes a bar code imprinted on theback of the page, opposite to the writing surface, wherein the bar codeindicates the form identity. A complementary bar code reader is thenprovided on the electronic board to read the bar code and identify theform. However, the process of adding a bar code to each page used withthe device undesirably adds printing expense and complexity since theproper bar code must first be determined to identify the type of formand then a double-sided print or copy must be made such that the barcode is imprinted on the back of the page. In addition, the bar codereader disadvantageously adds components to the device. Thus, thereexists a still further need for a pen enabled computing device havingthe capability of determining the type of form engaged therewith withoutrequiring additional components and special modification of the form toindicate the form type to the device.

In addition, another embodiment of the Flickinger device discloses asingle small printed box provided on the form that must be checked bythe user with the pen to indicate the form type. Accordingly, each formhas a separate box disposed in a different position about the writingsurface. Once a box is checked by the user to identify the particularform being used, the identification data corresponding to the marked boxmust be transmitted to another computing device to identify thecorresponding form type. However, having a dedicated box for each formtype, with each box being disposed in a dedicated position on thewriting surface, further limits the capacity of the device at theexpense of the available area of the writing surface. This limitation ismagnified if the form comprises multiple pages, wherein a separate boxmust be provided to identify each page of the form. In addition, thisform type indicating system does not provide for the identification ofmultiple pages of a single form type. Thus, there exists yet a furtherneed for a method of identifying a form to a pen enabled computingdevice that is capable of indicating form type while also uniquelyidentifying forms having multiple pages as well as multiple pages of asingle form type and while occupying minimal space on the writingsurface of the page. In some instances, it may be desirable for the penenabled computing device to be capable of identifying the form withoutrequiring dedicated identification boxes on the writing surface whichmust be specifically checked by the user.

Flickinger further discloses a device capable of receiving and storingpen input data which is then subsequently processed by a separatecomputing device. This device limitation may undesirably affectportability and the size of the device. Also, the combination of aremote processing device and a simple form identification process doesnot provide the capability for identifying a specific page on whichfuture revisions or additions to the entered data can be expediently andefficiently performed by way of the corresponding form on the electronicboard. The Flickinger form identification process has a furtherlimitation in that accessing of previously stored data is limited to ageneral form type and does not allow the user to access a more focuseddata set such as, for example, data input into a specific field on aspecific page of a certain type of form. More generally, Flickinger'sinability to specifically identify the page being used with theelectronic board may preclude a wide range of data manipulationprocesses that depend on the specificity and focus of a data set thatcan be sorted by a specific page identifier. Thus, there exists a needfor a pen enabled computing device capable of specifically identifyingthe page being used therewith in order to facilitate future datamodification, access, and manipulation in an efficient and expedientmanner to provide “seamless” operation with minimal inconvenience andenhanced flexibility for the user.

Thus, there exists a need for a pen enabled computing device having theinnate capability to process or otherwise manipulate recorded pen inputdata. It may also be desirable for the pen enabled computing device tohave the capability of detecting the presence of a page engaged with thedevice and to determine the type of form with a minimum of separateidentification devices and required user-initiated operations. Stillfurther, the pen enabled computing device should have the capability ofdetermining the type of form engaged therewith without requiringadditional components and special modification of the form to indicatethe form type to the device. In addition, it would be desirable to havean associated method of identifying a form to a pen enabled computingdevice that is capable of indicating form type as well as uniquelyidentifying forms having multiple pages and multiple pages of a singleform type, while occupying minimal space, if any, on the writing surfaceof the page. In some instances, it may also be desirable for the penenabled computing device to be capable of identifying the form withoutrequiring dedicated identification boxes on the writing surface whichmust be specifically checked by the user. Thus, there exists a need fora pen enabled computing device capable of specifically identifying thepage being used therewith in order to facilitate future datamodification, access, and manipulation in an efficient and expedientmanner to provide “seamless” and automatic operation with minimalinconvenience and enhanced flexibility for the user.

SUMMARY

The above and other needs are met by the present invention which, in oneembodiment, provides a system for identifying a preprinted form andinteracting therewith. The system comprises a pen enabled computingdevice having a writing stylus, a preprinted form, and a computerprogram product. The preprinted form is cooperable with the pen enabledcomputing device and comprises a page having a writing surface, a visualform identifier disposed on the writing surface and adapted to identifythe nature of the page to the user, and a plurality of fields defined bythe writing surface. The computer program product executes within thepen enabled computing device and cooperates therewith to determine thedisposition of the writing stylus adjacent to the writing surface inorder to actuate the pen enabled computing device. Subsequently, the penenabled computing device is capable of detecting, capturing, and storingdata input into the fields according to the nature of the page, whereinthe nature of the page comprises a function and a specific identity. Thenature of the page is also identifiable by the pen enabled computingdevice based on the data input into a plurality of the fields on thewriting surface.

In some instances, the system further comprises an electronichandwriting tablet capable of interacting with a pen enabled computingdevice and the preprinted form so as to sense the position and movementof the writing stylus with respect to a page engaged with thehandwriting tablet. In addition, the preprinted form may comprise aplurality of pages having a predetermined order such that a function anda specific identity may be determined for each page according to theorder. To facilitate determination of the nature of the page, the fieldsmay have predetermined spatial arrangement on the writing surfaceaccording to the nature of the page such that the pen enabled computingdevice detects and captures data input in a plurality of the fields todetermine the nature of the page. Alternatively, data may be input intoeach of a plurality of the fields, wherein the pen enabled computingdevice and the computer program product cooperate to process thedata-containing fields to define a corresponding base n number thatuniquely identifies the nature of the page to the pen enabled computingdevice. For example, the pen enabled computing device may be configuredto cooperate with the computer program product to define a base n numbercorresponding to a transposable matrix having n rows and m+1 columns,thereby forming an (n)*(m+1) matrix elements with each elementcorresponding to a field on the writing surface. Each matrix elementthereby has a value corresponding to the j*n^(i), wherein i may varyfrom 0 to m and j varies from 0 to n−1. Therefore, the base n number maybe, for example, the summation of the values of all of the elementswithin the matrix corresponding to field on the writing surface in whichthe user entered data. Once the nature of the page is determined, thepen enabled computing device may be further configured to enable accessto other data stored therein that is associated with the specificidentity of the page.

Once the nature of the page has been identified by the pen enabledcomputing device, data stored therein that is associated with and/orindependent of the specific identity of the page may be accessed. Insome instances, the data may be provided to the user via a userinterface. Still further, after the nature of the page has beendetermined, the pen enabled computing device cooperates with thecomputer program product to determine whether data input into the fieldscontinues to correspond to the determined nature of the page. The penenabled computing device accordingly indicates an error if the inputdata fails to correspond. If an error is indicated, the error isindicated to the user via the user interface. As such, the user maythereafter be prompted to correct the error through the pen enabledcomputing device. In some instances, the user interface is furthercapable of indicating to the user the fields containing data, the datacontained within the data-containing fields, and/or the number of fieldscontaining data.

The computer program product generally cooperates with the pen enabledcomputing device to detect, capture, and store handwritten data inputinto a field with the writing stylus. The handwritten data are thenspatially segmented such that the spatial segments correlate with thenature of the page. In some instances, the computer program product isfurther capable of assigning a stamp to data input into a field, whereinthe stamp corresponds to the nature of the page and may comprise a time,a date, and/or a unique identifier. Thus, in some cases, the nature of apage may comprise a function, a specific identity, and a stamp.

The pen enabled computing device may further comprise an editingactuator, wherein the computer program product is capable of cooperatingwith the pen enabled computing device to edit the data upon actuation ofthe editing actuator. When the editing actuator is actuated, thecomputer program product may cooperate with the pen enabled computingdevice to allow the addition of data to data contained within a field,to allow insertion of data between data elements contained within afield, to allow deletion of data contained within field, and/or to allowreplacement of data contained within field with alternate data. Thecomputer program product may also be capable of determining whether datais being input into a data-containing field having a preassigned stamp.If this is the case, the user is prompted, through the user interface,to verify that the user intends to edit the data within thedata-containing field.

For further manipulation of data, the computer program product may becapable of cooperating with the pen enabled computing device to allowthe exportation of data or importation of data in relation to anexternal computing device or within either of the pen enabled computingdevice and the external computing device. In such instances, thecomputer program product may be capable of reconciling the transmissionof data corresponding to the nature of a page between the pen enabledcomputing device and the external computing device, within the penenabled computing device, and/or within the external computing device.In such a system, the computer program product is capable of allowingthe pen enabled computing device and/or the external computing device toaccess data corresponding to the nature of the page and to examine andotherwise manipulate data contained within the fields thereof. In someinstances, the pen enabled computing device may cooperate with thecomputer program product to selectively direct data contained within afield to be converted into text by the pen enabled computing deviceand/or the external computing device upon actuation of a conversionactuator. Still more generally, the system may be capable of selectivelyconverting data contained within a field into text.

Another aspect of the present invention comprises a computer programproduct adapted to execute within a pen enabled computing device havinga writing stylus and to identify the nature of a preprinted form to thepen enabled computing device, wherein the form comprises a page having awriting surface defining a plurality of fields and the nature of thepage comprises a function and a specific identity. The computer programproduct generally comprises a number of executable portions executingwithin the pen enabled computing device. More specifically, the computerprogram product may comprise a first executable portion for determiningthe disposition of the writing stylus adjacent to the writing surfacesuch that the writing stylus is capable of interacting with the fieldsin order to actuate the pen enabled computing device. A secondexecutable portion directs the pen enabled computing device to detect,capture, and store data input into the fields with the writing stylus,wherein the data is input according to the nature of the page. A thirdexecutable portion identifies the nature of the page to the pen enabledcomputing device based upon data input into a plurality of the fields.

In some instances, the computer program product is adapted to facilitateinteraction between the pen enabled computing device and an electronichandwriting tablet capable of sensing position and movement of thewriting stylus with respect to a page engaged therewith. Where the formcomprises a plurality of pages having a predetermined order, thecomputer program product may further comprise a fifth executable portionfor determining a function and a specific identity for each pageaccording to the order. Further, the fields may have a predeterminedspatial arrangement on the writing surface according to the nature ofthe page such that the second executable portion of the computer programproduct detects and captures data input in a plurality of the fields toenable the pen enabled computing device to determine the nature of thepage. Alternatively, data may be input into each of a plurality of thefields, wherein the computer program product then processes thedata-containing fields to define a corresponding base n number thatuniquely identifies the nature of the page to the pen enabled computingdevice. For example, the pen enabled computing device may be configuredto cooperate with the computer program product to define a base n numbercorresponding to a transposable matrix having n rows and m+1 columns,thereby forming an (n)*(m+1) matrix elements with each elementcorresponding to a field on the writing surface. Each matrix elementthereby has a value corresponding to the j*n^(i), wherein i may varyfrom 0 to m and j varies from 0 to n−1. Therefore, the base n number maybe, for example, the summation of the values of all of the elementswithin the matrix corresponding to field on the writing surface in whichthe user entered data.

After the nature of the page has been determined, the fourth executableportion may also access data stored within the pen enabled computingdevice, wherein the data may be with and/or be independent of thespecific identity of the page. Where the pen enabled computing devicefurther comprises a user interface, the computer program product furthercomprises a sixth executable portion adapted to cooperate with the userinterface to provide data to the user. The computer program product mayalso comprise a seventh executable portion for determining whether datainput into the fields continues to correspond to the determined natureof the page and indicating an error if the input data fails tocorrespond. In instances where an error is indicated to the user throughthe user interface, the seventh executable portion may also prompt theuser to correct the error through the pen enabled computing device. Inother instances, the seventh executable portion may be capable ofindicating to the user through the user interface the fields containingdata, the data contained within the data-containing fields, and/or thenumber of fields containing data.

Further, the second executable portion of the computer program productmay detect, capture, and store handwritten data input into a field withthe writing stylus by spatially segmenting the handwritten data andcorrelating the spatial segments with the nature of the page. In someinstances, the computer program product may further comprise an eighthexecutable portion for assigning a stamp to data input into a field,wherein the stamp corresponds to the nature of the page and may comprisea time, a date, and/or a unique identifier.

Once a page is sufficiently identifiable, such as by a function,specific identity, and/or stamp, the data may be accessed and otherwisemanipulated. In some instances, the pen enabled computing device maycomprise an editing actuator, wherein the computer program productfurther comprises a ninth executable portion for enabling editing ofdata upon actuation of the editing actuator. The ninth executableportion may also be capable of adding data to data contained within afield, inserting data between previous data elements contained within afield, deleting data contained within a field, and/or replacing datacontained within a field with alternate data. Also, the computer programproduct may further comprise a tenth executable portion capable ofdetermining whether data is being input into a data-containing fieldhaving a preassigned stamp and, if so, prompting the user, through theuser interface, to verify that the user intends to edit the data withinthe field.

The computer program product may further comprise an eleventh executableportion capable of exporting data to and/or importing data in relationto an external computing device or within either the pen enabledcomputing device or the external computing device. The eleventhexecutable portion may be capable of reconciling transmission of datacorresponding to the nature of the page between the pen enabledcomputing device and the external computing device, within the penenabled computing device, and/or within the external computing device.The computer program product may further comprise a twelfth executableportion capable of allowing the pen enabled computing device and/or theexternal computing device to access data corresponding to the nature ofa page and to examine and otherwise manipulate data contained within thefields thereof. In some instances, the pen enabled computing device mayfurther comprise a conversion actuator, wherein the computer programproduct also further comprises a thirteenth executable portion that iscapable of selectively directing data contained within a field to beconverted into text by the pen enabled computing device and/or theexternal computing device upon actuation of the conversion actuator.Still more generally, the computer program product may further comprisea fourteenth executable portion capable of selectively converting datacontained with a field into text.

Yet another aspect of the present invention comprises a pen enabledcomputing device adapted for use with a preprinted form, the preprintedform comprising a page having a writing surface, a visual formidentifier disposed on the writing surface and adapted to identify thenature of the page to a user, and a plurality of fields defined by thewriting surface and arranged to correspond to the nature of the page,further wherein the nature of the page generally comprises a functionand a specific identity. Such a pen enabled computing device comprises awriting stylus and a sensing device, wherein the sensing device isadapted to engage the preprinted form and cooperates with the writingstylus such that the pen enabled computing device is actuated when thewriting stylus is disposed adjacent to the writing surface ininteractable relation with the fields, thereby enabling the pen enabledcomputing device to determine the nature of the page as data is inputinto the fields with the writing stylus according to the user-identifiednature of the page.

The sensing device may comprise an electronic handwriting tablet adaptedto engage the preprinted form and capable of sensing position andmovement of the writing stylus with respect to the page. In someinstances, the pen enabled computing device may be configured to becapable of determining the nature of a form comprising a plurality ofpages having a predetermined order where each page has a function andspecific identity corresponding to the order. In order to determine thenature of the page, the pen enabled computing device may be configuredto require data to be input into a plurality of the fields in order todetermine the nature of the page therefrom, wherein the fields have apredetermined spatial arrangement on the writing surface according tothe nature of the page. Alternatively, the pen enabled computing devicemay be configured to define a base n number that uniquely identifies thenature of the page from the data input into each of a plurality of thefields. For example, the pen enabled computing device may be configuredto cooperate with the computer program product to define a base n numbercorresponding to a transposable matrix having n rows and m+1 columns,thereby forming an (n)*(m+1) matrix elements with each elementcorresponding to a field on the writing surface. Each matrix elementthereby has a value corresponding to the j*n^(i), wherein i may varyfrom 0 to m and j varies from 0 to n−1. Therefore, the base n number maybe, for example, the summation of the values of all of the elementswithin the matrix corresponding to field on the writing surface in whichthe user entered data. Once the nature of the page is determined, thepen enabled computing device may be further configured to enable accessto other data stored therein that is associated with the specificidentity of the page.

A still further aspect of the present invention comprises a method foridentifying the nature of a page of a preprinted form with a pen enabledcomputing device having a writing stylus and interacting therewith.Generally, the nature of the page comprises a function and a specificidentity, wherein the page further has a writing surface, a visual formidentifier disposed on the writing surface and adapted to identify thenature of the page to a user, and a plurality of fields defined by thewriting surface. Such a method comprises determining the disposition ofthe writing stylus adjacent to the writing surface such that the writingstylus is capable of interacting with the fields and actuating the penenabled computing device, receiving data into the fields via the writingstylus and according to the nature of the page, determining the natureof the page from a plurality of the fields in which data is received,and identifying the nature of the page to the pen enabled computingdevice.

In some instances, the method may further comprise allowing access toother data stored within the pen enabled computing system, wherein thedata may be associated with and/or independent of the specific identityof the page. Where the form comprises a plurality of pages having apredetermined order, the method may also further comprise determining afunction and specific identity for each page corresponding to the order.In some cases, the fields may be defined by the writing surfaceaccording to a predetermined spatial arrangement, wherein the methodthen further comprises determining the nature of the page from thespatial arrangement of the plurality of the fields in which data isreceived. Alternatively, the method may comprise determining the natureof the page from a plurality of the fields in which data is received,wherein the data input into each of the plurality of the fields isprocessed to define a base n number uniquely corresponding to the natureof the page. Accordingly, the input data may be determined by sensingthe position and movement of a writing stylus with respect to a pagewith an electronic handwriting tablet engaged with the page.

The method may also comprise providing data to the user via a userinterface, wherein the user interface is cooperable with the pen enabledcomputing device. After the nature of the page has been identified tothe pen enabled computing device, the method may also comprisedetermining whether data input into the fields continues to correspondto the determined nature of the page and indicating an error if theinput data fails to correspond. In such instances, the method mayfurther comprise indicating an error to the user through the userinterface and prompting the user to correct the error using the penenabled computing device. The user interface may also be configured suchthat the method may further comprise indicating to the user the fieldscontaining data, the data contained within the data-containing fields,and/or the number of fields containing data.

In addition, the method may comprise detecting, capturing, and storinghandwritten data input into a field with the writing stylus by spatiallysegmenting the handwritten data and correlating the spatial segmentswith the nature of the page. In some instances, the method may alsocomprise assigning a stamp to data input into a field, wherein the stampcorresponds to the nature of the page and comprises a time, a date,and/or a unique identifier. Where data may be edited, the method mayfurther comprise actuating an editing actuator in order to enable thepen enabled computing device to edit the data. After actuating theediting actuator, the pen enabled computing device may also be enabledto add data to data contained within a field, insert data between dataelements contained within a field, delete data contained within a field,and replace data contained within a field with alternate data. In someinstances, the method may further comprise determining whether data isbeing input into a data-containing field having a preassigned stamp and,if so, prompting the user through the user interface to verify that theuser intends to edit the data within the field.

Still further, detecting the position and movement of the writing stylusmay further comprise sampling the motion of the writing stylus withrespect to the writing surface so as form a coordinate representation ofeach stroke of the handwritten input. In some instances, editing thehandwritten input may comprise receiving an indication from a user thatthe handwritten input is to be edited and receiving textual input from akeyboard to form the textual data with which the handwritten input isedited. Accordingly, editing the handwritten input may further comprisealtering the coordinate representation of at least some strokes of thehandwritten input in order to alter spacing of the at least one boundingbox so as to insert the textual input between adjacent bounding boxesand/or replace a bounding box containing handwritten input with thetextual input.

Yet another advantageous aspect of the present invention comprises amethod for converting handwriting into a machine-usable format using apen enabled computing device having a writing surface and a writingstylus selectively communicable with the writing surface. First, theposition and movement of the writing stylus with respect to the writingsurface is detected by the pen enabled computing device to form ahandwritten input. Bounding boxes are then defined around each stroke ofthe handwritten input. The handwritten input and the bounding boxes arethen stored in a machine-usable format, thereby allowing the handwritteninput to be edited by adding textual data to the handwritten input,inserting textual data between adjacent bounding boxes, deleting atleast one bounding box, and replacing at least one bounding box withalternate data.

Still further, detecting the position and movement of the writing stylusmay further comprise sampling the motion of the writing stylus withrespect to the writing surface so as form a coordinate representation ofeach stroke of the handwritten input. In some instances, editing thehandwritten input may comprise receiving an indication from a user thatthe handwritten input is to be edited and receiving textual input from akeyboard to form the textual data with which the handwritten input isedited. Accordingly, editing the handwritten input may further comprisealtering the coordinate representation of at least some strokes of thehandwritten input in order to alter spacing of the at least one boundingbox so as to insert the textual input between adjacent bounding boxesand/or replace a bounding box containing handwritten input with thetextual input.

Yet still another advantageous aspect of the present invention comprisesa system for converting handwriting into a machine-usable format,wherein the system comprises a pen enabled computing device having amemory element, a writing surface, and a writing stylus selectivelycommunicable with the writing surface. The system may also comprise acomputer program product executing within the pen enabled computingdevice and cooperating therewith for detecting the position and movementof the writing stylus with respect to the writing surface to form ahandwritten input, defining a bounding box about each stroke of thehandwritten input, storing the handwritten input and the bounding boxsurrounding each stroke in the memory element in a machine-usableformat, and editing the at least one handwritten element to add textualdata to the handwritten input, insert textual data between adjacentbounding boxes, delete at least one bounding box, and/or replace atleast one bounding box with alternate data.

According to some embodiments, the computer program product may samplethe motion of the writing stylus with respect to the writing surface soas to form a coordinate representation of each stroke of the handwritteninput. Further, the pen enabled computing device may cooperate with thecomputer program product to receive an indication from a user of thehandwritten input to be edited and also receive textual input from akeyboard to form the data with which the handwritten input is edited.Accordingly, the computer program product edits the handwritten input byaltering the coordinate representation of at least some strokes of thehandwritten input in order to alter spacing of the at least one boundingbox so as to insert the textual input between adjacent bounding boxesand/or replace a bounding box containing a handwritten element with thetextual input.

Thus, embodiments of the present invention provide a pen enabledcomputing device having the innate capability of processing or otherwisemanipulating recorded pen input data. Embodiments of the presentinvention also provide a pen enabled computing device having thecapability of detecting the presence of a page engaged with the deviceand determining the type of form with a minimum of separateidentification devices and required user-initiated operations. Accordingto a particularly advantageous aspect, the user uses the form accordingto its identified function and does not otherwise have to independentlyindicate the nature of the form to the pen enabled computing device.Accordingly, embodiments of the present invention provide a pen enabledcomputing device having the capability of determining the type of formengaged therewith without requiring additional components or specialmodification of the form to indicate the type of form to the device.Embodiments of the present invention also provide an associated methodof identifying a form to a pen enabled computing device, whileindicating the type of form as well as uniquely identifying forms havingmultiple pages and multiple pages of a single type of form.Advantageously, such a method accomplishes the identification of thepage to the pen enabled computing system while occupying minimal, ifany, space on the writing surface of the page. Thus, embodiments of thepresent invention provide a pen enabled computing device capable ofspecifically identifying the page being used therewith in order tofacilitate future data modification, access, and manipulation in anefficient and expedient manner to provide enhanced seamless andautomatic operation with minimal inconvenience and enhanced flexibilityfor the user.

BRIEF DESCRIPTION OF THE DRAWINGS

Some of the advantages of the present invention having been stated,others will appear as the description proceeds, when considered inconjunction with the accompanying drawings, which are not necessarilydrawn to scale, in which:

FIG. 1 is a schematic of a pen enabled computing system according to oneembodiment of the present invention.

FIG. 2 is a schematic of a pen enabled computing device according to oneembodiment of the present invention.

FIG. 3 is an example of a contacts form in accordance with oneembodiment of the present invention.

FIG. 4 is an example of a daily schedule form in accordance with oneembodiment of the present invention.

FIG. 5 is an example of a to-do list form in accordance with oneembodiment of the present invention.

FIG. 6 is an example of a notes form in accordance with one embodimentof the present invention.

FIGS. 7 (a) and 7 (b) is a flowchart of a method of identifying a formand manipulating data associated therewith with a pen enabled computingdevice according to one embodiment of the present invention.

FIG. 8 is a view of a handwritten input in accordance with one exampleembodiment.

FIG. 9 is another view of the handwritten input of FIG. 8.

DETAILED DESCRIPTION

The present invention now will be described more fully hereinafter withreference to the accompanying drawings, in which preferred embodimentsof the invention are shown. This invention may, however, be embodied inmany different forms and should not be construed as limited to theembodiments set forth herein; rather, these embodiments are provided sothat this disclosure will be thorough and complete, and will fullyconvey the scope of the invention to those skilled in the art. Likenumbers refer to like elements throughout.

FIG. 1 is a schematic of a system for identifying a form andmanipulating associated information with a pen-enabled computing deviceaccording to one embodiment of the present invention, the device beingindicated generally by the numeral 100. The device 100 generallycomprises a handwriting capture interface 200, a central processingsubsystem 300, a user interface 400, a computer program product 500, andan external device interface 600. The handwriting capture interface 200generally comprises an apparatus for sensing the movements of a writingstylus. In this regard, the handwriting capture interface typicallyprovides a digitized output that defines a number of (x,y) coordinatepairs, each of which defines a mark made by the writing stylus upon awriting surface. The writing stylus is typically a writing instrumentwhich may be capable of creating visible marks upon a writing surfacesuch as, for example, a blank of piece of paper 210 or a paper-basedpreprinted form 220. In a particularly advantageous embodiment of thepresent invention, the handwriting capture interface 200 comprises awriting stylus and an electronic handwriting tablet area. The stylus andthe handwriting tablet are interfaced such that the position or motionof the writing stylus, corresponding to writing strokes, with respect tothe handwriting tablet can be detected and captured by the device 100.The stylus may interact with the handwriting tablet such that theposition or motion of the stylus is detected when proximate to thetablet or, alternatively, in contact with the handwriting tablet througha writing surface, wherein the writing surface may comprise, forexample, the paper 210 or form 220, a multiplicity thereof, or a surfaceintegral with the handwriting tablet. In some instances, a handwritingtablet may not be necessary if the position and motion of the writingstylus may be directly detected by the handwriting capture interface 200or other components of the device 100.

At least partially driving the handwriting capture interface 200 is thecentral processing subsystem 300. The central processing subsystem 300may comprise, for example, a central processor unit 320 , non-volatilememory storage 340 , and RAM storage 360. The central processing unit(CPU) 320 is generally capable of executing all computer processingactivities required to support the device 100. Non-volatile memoryresources 340 such as, for example, an EEPROM or a disk, are typicallyused for storing and/or executing a computer program product 500 as wellas storing useful data such as, for example, electronic ink. Further,the RAM storage 360 is used for loading and executing the computerprogram product 500.

The user interface 400 includes provisions for facilitating interactionbetween the device 100 and a user. The user interface 400 may comprise,for example, one or more buttons for initiating functions within thedevice 100 where, in some instances, the buttons may comprise a part ofa keyboard or a keypad. The user interface 400 may also comprise anapparatus for allowing the user to view displayed data or statusinformation such as, for example, a viewing screen or monitor. In someinstances, the user interface 400 may also include an audio mechanismsuch as, for example, a buzzer, bell, or other indicator or otherdevices capable of supporting voice interaction between the device 100and the user. Still further, for example, a vibration mechanism may alsobe used as an alert or acknowledgement indicator. Accordingly, the userinterface 400 may be implemented in many different manners, combinationsthereof, and/or in conjunction with different components of the device100.

The computer program product 500 generally cooperates with the centralprocessing subsystem 300 to control the device 100. The computer programproduct 500 further drives the handwriting capture interface 200 tointerface and cooperate with a writing surface. According to oneembodiment, the computer program product 500 enables the device 100 to,for example, sense a plurality of fields associated with the paper 210or the preprinted form 220, wherein the function and specific identitythereof may then be electronically identified to the device 100. Thefunction and specific identity of the form may also be referred toherein as comprising the nature of the form. While the handwritingcapture interface 200 may be implemented with, for example, a piece ofpaper 210, a form 220, or a surface integral with the interface 200,reference to the handwriting capture interface 200 will be made hereinto an embodiment of the invention having a form 220 interactingtherewith. It will be understood, however, that this reference is forillustration only and such reference may include embodiments, otherwisementioned herein as examples, as well as other configurations of ahandwriting capture device consistent with the spirit and scope of thepresent invention.

Once the device 100 interfaces with the form 220 based upon theidentified function and specific identity thereof, the system is thenable to act upon data entered in the fields associated with the form,the data comprising, for example, a handwriting input. In oneadvantageous embodiment of the present invention, the handwriting inputis captured and stored by the computer program product 500 as acoordinate representation of the interaction of the writing stylus withthe handwriting tablet. In particular, the handwriting capture interfaceprovides a plurality of (x,y) coordinate pairs representative torespective marks made by the writing stylus upon the handwriting tablet.As described hereinafter, the central processing subsystem defines abounding box around the (x,y) coordinate pairs that cumulativelyrepresent a particular stroke. In this context, a stroke is generallydefined as a continuous marking by the writing stylus beginning with thecommencement of contact or other interaction between the writing stylusand the handwriting tablet and terminating with the removal of thewriting stylus from the handwriting tablet. For example, a “c” isgenerally formed of one stroke, while a “t” is generally formed of twostrokes. Once the nature of the form is determined from the input data,the computer program product 500 is capable of accessing other data thatis associated and/or not associated with the specific identity of theform. In some instances, the data in the fields may also be altered orotherwise manipulated by the user. The computer program product 500 mayfurther have the ability to stamp any data entries as they are enteredinto the system 100 with, for example, a time, a date, and/or anotherunique identifier. The device 100 is then able to process these datafields with respect to the stamp. As these functions are executed, thecomputer program product 500 also controls the user interface 400 suchthat information relevant to ongoing stylus or device 100 activities aredisplayed for the user. In one advantageous embodiment, the computerprogram product 500 may be capable of formatting, sending, receiving,reconciling, and interpreting messages flowing through the externaldevice interface 600. The computer program product 500 is described inmore detail herein.

The external device interface 600 is in communication with the centralprocessing subsystem 300 and enables the device 100 to communicate withexternal computing devices in a unidirectional or bidirectional manner.Generally, the communication is accomplished via a communications link700 to an external computing device 800. Accordingly, the externaldevice interface 600 may be generic in terms of protocol andcommunication medium as is known to one skilled in the art. Thecommunications link between the device 100 and the external computingdevice 800 may be either direct or indirect connections via wireline orwireless devices. The communications link 700 may comprise a variety ofexternal communications links as well as a combination of these linkssuch as, for example, wired serial interfaces, wired parallelinterfaces, wireline modems, wireless modems, LAN connections, and WANconnections.

One advantageous embodiment of a pen-enabled computing device 100according to the present invention is shown in FIG. 2. A handheldcomputer component 110 such as, for example, a modified personal digitalassistant (PDA) or a smart-phone provides most of the functionality ofthe device 100. Accordingly, the handheld computer component 110comprises the central processing subsystem 300, the user interface 400,the computer software program product 500, and the external deviceinterface 600 as shown in FIG. 1. The user interface 400 may include aninteractive display 120, one or more control buttons 130, and a speaker140. The external communications interface 600 may include a wirelesstransceiver (not shown) and an antenna 150. In addition, the handwritingcapture interface 200 comprises an electronic handwriting tablet 230that may be co-operable with, for example, a blank piece of paper 210 ora paper-based preprinted form 220. The electronic handwriting tablet 280may be serially interfaced with the handheld computer component 110. Adual-mode writing stylus 290 is employed by the user to form visiblemarks on the paper-based preprinted form 220, wherein the motion andposition of the writing stylus 290 are also detectable by the electronichandwriting tablet 280.

A pen-enabled computing device 100 according to the present inventionmay comprise many different combinations of components than theembodiment previously described. For example, an electronic handwritingtablet may be combined with a desktop personal computer, wherein theelectronic handwriting tablet and the personal computer are physicallyconnected via a wired interface and the personal computer is connectedto a local area network (LAN). As another example, an electronichandwriting tablet may be combined with a laptop personal computer and acentral server, wherein the electronic handwriting tablet and thepersonal computer are connected via a wireless interface and thepersonal computer has a network connection to the central server via aLAN. The central server is further capable of communicating with variousapplication servers. Thus, a pen-enabled computing device may beaccomplished in many different manners consistent with the spirit andscope of the present invention.

In accordance with embodiments of the present invention, one purpose ofthe pen enabled computing device is to identify the nature of a form 220engaged therewith. Several examples of forms are shown in FIGS. 3-6,wherein such forms 220 may include a contact list form as shown in FIG.3, a calendar form as shown in FIG. 4, a to-do list form as shown inFIG. 5, and a general notes form as shown in FIG. 6. Other examples offorms include, for instance, a trip planner form, a memorandum form, aneducational form, a training form, an insurance form, a healthcare form,an expense accounting form, and a wide variety of other forms consistentwith the spirit and scope of the present invention.

As shown in FIG. 3, a particular contacts form 220 comprises a writingsurface 222, at least one visual form identifier 224, and a plurality offields 226 for receiving data via the writing stylus. The fields 226 aregenerally defined by a series of spatial coordinates so as to generallydefine a box configured to receive a data input therein corresponding toa particular function. For example, a field labeled “Address” is capableof identifying itself to a user and is expected to receive an inputtherein with the writing stylus corresponding to the address of aparticular contact. Further, fields labeled, for example, “G” and “7”could indicate that the user wishes to store an entered record on theseventh page of all contacts having data within the “Name” field havingthe letter “G”. Thus, for a given type of form 220, the fields 226defined by the writing surface 222, along with the visual formidentifiers 224, may combine to indicate a function of the form as wellas a specific identity should the form comprise multiple pages or shouldthere be multiple pages of one type of form.

Generally, the user interacts with the form 220 according to theuser-identified function imparted by the visual form identifiers 224.The user then enters data within the separate fields according to thefunctions thereof. Typically, as the data is entered into a field, it isspatially referenced with respect to the box defining the field. Forexample, data entered into a “Name” field may be referenced to theleftmost vertical boundary or possibly a combination of the leftmostvertical boundary and the lowermost horizontal boundary. Thereafter,data entered into an “Address” field may be referenced to the leftmostvertical boundary of the address box, which also corresponds to therightmost vertical boundary of the name field. Thus, it is understoodthat data entered into defined fields on a form according to thefunctions thereof may be spatially referenced so as to be indicative ofthe nature of the form 220.

FIG. 4 shows an example of a daily scheduler form 228 which alsoincludes a writing surface 230, at least one visual form identifier 232,and a plurality of fields 234 defined by the writing surface 230. Thefields 234 of the daily scheduler form 228 are spatially defined in adifferent manner than the previous contacts form 220 as shown in FIG. 3.Thus, the fields for indicating the month and day in the daily schedulerform 228 may fall outside any of the fields defined by the contacts form220. Conversely, certain fields in the contacts form 220 , such as the“Address” or “E-mail” fields may be spatially disposed away from thespatial reference points for the fields of the daily scheduler form 228.Thus, it will be understood that spatial representation of fieldsdisposed on a form can also serve to differentiate between differenttypes of forms. In order to add more specificity to the identificationof a form, various fields may also be spatially disposed so as to definea unique code over other forms. For example, checking the appropriatefields on the daily scheduler form 228 for a month and daydifferentiates the particular form from daily scheduler forms for otherdays throughout the year.

As shown in FIG. 2, the pen enabled computing device 100 is particularlyadapted to interact with a form 220, wherein entering data in fields onthe form 220 with a writing stylus 240 provides both a written ink copyof the particular form in addition to an electronic ink copy detected,captured, and stored by the device 100. In order to facilitate futureaccess to and other manipulation of the data entered on a particularform 220, the device 100 is configured to be capable of identifying thenature of the form 220, wherein the nature of the form comprises afunction and a specific identity. According to one embodiment of thepresent invention, the nature of the form 220 may be defined by a codecorresponding to data entered into particular fields on the form 220. Onthe writing surface of the form, the particular fields may be disposedand configured so as to form a unique code depending on the particularfields in which data is entered. For example, the pen enabled computingdevice 100 may be able to define a base n number that indicates thenature of a particular form. For example, the pen enabled computingdevice may be configured to cooperate with the computer program productto define a base n number corresponding to a transposable matrix havingn rows and m+1 columns, thereby forming an (n)*(m+1) matrix elementswith each element corresponding to a field on the writing surface. Eachmatrix element thereby has a value corresponding to the j*n^(i), whereini may vary from 0 to m and j varies from 0 to n−1. Therefore, the base nnumber may be, for example, the summation of the values of all of theelements within the matrix corresponding to field on the writing surfacein which the user entered data. Thus, it can be seen that selecting aparticular combination of matrix elements within the matrix provides aunique identifier which may be used to indicate the nature of the formand such defined coding may be appropriately scaled or otherwise alteredto match the form type support capacity of the device 100. In thismanner, the area occupied on the writing surface by the particularfields may be reduced or minimized since the number of fields requiredto produce a unique form identification code requires less area on thewriting surface compared to specifically dedicated identificationfields.

In addition, various other provisions can be used to enhance and/orsupplement the nature of the form indicated by a particular combinationof actuated fields. For example, the pen enabled computing device 100may be configured to selectively apply a stamp to data entered within aparticular field, wherein the stamp may comprise a time, a date, or someother type of unique identifier that may be further referenced by thepen enabled computing device 100. Alternatively, or in addition, the penenabled computing device 100 may be configured so as to be capable ofidentifying the nature of the form based upon the spatial disposition ofthe fields defined by the writing surface. According to suchembodiments, the pen enabled computing device 10 is capable ofdetermining the spatial disposition of data entered on the writingsurface in comparison to the spatial references for particular fieldsfor the various forms supported by the pen enabled computing device 100.That is, fields on different forms can be staggered or otherwiseuniquely positioned with respect to other forms supported by the systemso that each form has a unique combination of fields that can beautomatically identified by the pen enabled computing system as the userenters data in one or more of the fields. In some instances, theidentification of a unique combination of fields may be sufficient toindicate the nature of the form to the device 100.

The enhanced determination of the nature of the form according toembodiments of the present invention provides further utility in thatfuture editing of the input data is facilitated as well as futureediting of the particular paper-based form. In addition, error detectionand correction are also facilitated. For example, according to oneembodiment of the present invention, once the nature of the page hasbeen defined by the device 100, data entered according to a particularspatial arrangement of fields is expected by the device 100.Accordingly, should the user enter data outside the defined fieldboundaries, the device 100 is capable of, for example, detecting whetherthe user has entered data outside of an expected defined field and mayapply a spatial criteria to determine whether the data has beeninsignificantly entered outside the boundary of the field by the user.If the entered data is determined by the device 100 to have been enteredby the user outside the expected fields corresponding to the nature ofthe page, the device 100 may alert the user with an alarm or some otheruser interface device and present the user with the opportunity toresolve the aberrant data. For example, the user could be asked toidentify to the device 100 the user-identified nature of the page.Alternatively, the user could be presented with the opportunity to movethe aberrant data to an expected field or given the opportunity to savethe aberrant data as an incidental note not associated with a particularexpected field. Accordingly, it is understood that such applied criteriamay facilitate a variety of possibilities for resolving the misentereddata as well as other data processing operations. According to otherembodiments of the present invention, detailed identification of thenature of the page facilitates the indication of data to the user as theuser is interacting with the device 100. For example, upon determiningthe nature of the form, the device could display to the user anyprevious data entered into the corresponding fields, the nature of thatdata, or possibly the number of fields in which the data has beenentered. Accordingly, the device 100 may also be able to determinewhether the user is entering data in a field in which data had beenpreviously entered. In detecting the situation, the device 100 mayfurther be capable of alerting the user as to the presence of theprevious data as well as given the user the opportunity to add to thedata in the field, replace the data in the field, or abort the input ofdata into that particular field.

FIGS. 7A and 7B provide a flowchart of a method of identifying a formand manipulating information using a pen enabled computing deviceaccording to one embodiment of the present invention. Where the form isengaged with the pen enabled computing device 100 and the user proceedsto enter data within the fields on the writing surface, the device 100determines the position and senses the movement of the writing styluswith respect to the writing surface (block 1000) and a text, receives,and stores the data entered into the fields of the page (block 1010). Ifthe user is entering data into a field defined by a previous stamp, thestamp comprising a time, date, or other unique identifier (block 1020),the device 100 prompts the user to indicate whether the user wishes toproceed with editing the existing field (block 1030). If the userindicates that no editing of the current field is desired, the device100 indicates an error message to the user (block 1040) and allows theuser to select an alternate field on the writing surface or, forexample, resolve a conflict in form types. However, if the userindicates that editing the entry is desired (block 1050), the device 100allows the user to edit the data and then saves the edited data andassigns a new stamp to the field (block 1060). Alternatively, if thefield had contained no previous stamp, the user would be permitted toinput new data into the field which would be assigned a stamp followingthe completion of data entry by the user (block 1070).

After the data has been input into the field and a stamp assigned by thedevice 100, the data is stored by the device 100 (block 1080). Dependingon a variety of factors including the configuration of the device 100and the type of form, the device 100 may then proceed to determine thespatial arrangement of the fields in which data was entered by the user(block 1090) and/or proceed to define a base n number based on thefields having a data input (block 1100). The nature of the particularpage can then be determined (block 1110) and the device 100 is then ableto verify that the data entered into the selected fields corresponds tothe identified nature of the page (block 1120). Should the data notcorrespond to the nature of the page, the device indicates an errormessage to the user (block 1130) and further prompts the user to selecta different field or resolve the conflict between form types. However,should the data input into the fields correspond to the nature of thepage, the device 100 allows access to other data based on the nature ofthe page (block 1140) or according to a more focused search criteriaspecified by the user. Further, data including the data input by theuser, the data contained within fields within the form, and other dataassociated with the nature of the particular page may also be providedto the user through the user interface (block 1150) such that the useris able to monitor the data status and to edit or otherwise manipulatethe data associated with the particular page. However, it should benoted that, once the data has been captured and stored by the device100, that a variety of data manipulation functions are possible usingthe device 100 and/or an external computing device 800 according to thespirit and the scope of the present invention.

With reference to FIGS. 8 and 9, the present invention also provides atechnique for seamlessly editing handwritten and/or textual data bymeans of a pen-enabled computing device 100 and a computer programproduct 500 executing thereon. In the embodiment in which a handwritteninput 899 is to be edited, the handwritten input is initially capturedby the handwriting capture interface 200 which detects the position andmovement of the writing stylus 290 with respect to the writing surface280 of the pen-enabled computing device 100. As described above, thehandwriting capture interface 500 generally provides a sequence of (x,y)coordinate pairs 900 that define respective marks by the writing stylus290 upon the writing surface 280.

The central processing unit 320 of the pen-enabled computing device 100in conjunction with the computer program product 500 then definebounding boxes 902 around each stroke 904. Typically, each bounding box902 is a rectangular box 902 that includes at least a portion of the(x,y) coordinate pairs 900 of the respective stroke 904 with one xcoordinate of the rectangular box 902 defined by the smallest x 900 a ofany of the coordinate pairs 900 and another x coordinate of therectangular box 902 at a point defined by the largest x 900 b, with they coordinates extending between successive baselines B1, B2, . . . Bn.In some instances, the bounding boxes 902 may also have included in thedefinitions thereof a time variable which may affect the delegation of astroke to a particular bounding box. While a bounding box 902 isdescribed herein in spatial terms as at least partially encompassing asingle stroke 904, the boundary box 902 can be defined differently inother applications, typically based on spatial or temporalcharacteristics of one or more strokes 904 in order to bundle one ormore strokes 904 together as a unit 910 for editing purposes.

In instances in which the handwritten input 899 is arranged in multiplelines, the central processing unit 320 in cooperation with the computerprogram product 500 operating thereon also determines the baseline Bnfor each of the lines so as to provide a point of reference for thecharacters in the respective line. As known to those skilled in the art,the baseline Bn typically corresponds to the lowermost portion of mostalphanumeric characters, although several letters, such as p, g and jhave lower portions that fall below the baseline Bn. The baseline Bn canbe determined in various manners. For example, the user of thepen-enabled computing device 100 can place the baseline Bn by dragging abaseline Bn into the proper position, by positioning a baseline Bn withrepeated actuation of up and down arrow keys or any other like means. Asknown to those skilled in the art, baselines Bn can also be determinedbased upon changes in a one-dimensional histogram in which the histogramaxis corresponds to the y or vertical direction of the page that theuser is writing upon. In this regard, the histogram includes counts fora number of y values that define the number of strokes 904 that passthrough the respective y value. Since the y values that have largecounts generally correspond to lines of handwriting, a baseline Bn istypically defined immediately below y values having large counts.Finally, if a lined page is placed upon the writing surface 280 suchthat the page 220 is positioned in a known registration to the writingsurface 280, the central processing unit 320 can define the baselines Bnin a spatial arrangement corresponding to the spatial arrangement oflines on the page 220.

Regardless of the manner in which the baselines Bn are determined, thehandwritten input 899 can be edited by initially displaying thehandwritten input 899 to the user upon a display 400 associated with thepen-enabled computing device 100. The user then selects an actuationposition 910 relative to the handwritten input 899, such as bypositioning the a cursor 912 through the use of a mouse, stylus, a trackball or the like, relative to the display of the handwritten input 899.Depending upon the location of the cursor relative to the bounding boxes902 that surround each stroke 904, the pen-enabled computing device 100may shift the cursor 912 position somewhat. In this regard, the centralprocessing unit 320, in cooperation with the computer program product500, determines if the cursor 912 position is located within a boundingbox 902. If the cursor 912 position is located within one or morebounding boxes 902, the pen-enabled computing device 100 determines thex coordinate of the center of the largest bounding box 902. If thecursor position is at an x position greater than the x position of thecenter of the largest bounding box, the central processing unit 320shifts the cursor 912 position to correspond to or be slightly largerthan the largest x value 900 b associated with the bounding box 902,i.e., shifts the cursor 912 to the right side of the largest boundingbox. Alternatively, if the cursor 912 position has an x value that isless than the x position of the center of the largest bounding box 902,the central processing unit 320 shifts the cursor 912 position to an xposition that is equal to the smallest x value 900 a associated with thebounding box 902, i.e., shifts the cursor 912 position to the left ofthe smallest bounding box 902. If, however, the cursor 912 position isnot within a bounding box 902, the cursor 912 position need not beshifted whatsoever.

The pen-enabled computing device 100 then determines the editingfunction based upon the user input. If text is being inserted, forexample, the text may be input by means of keystrokes on the keypad orby a handwritten insert accomplished by further movement of the writingstylus 290 over the writing tablet 280 as described above. If text isbeing inserted, such as by means of the keypad, the strokes that definethe handwritten input 899 to the right of the insertion are movedfurther to the right by an amount equal to the size, in the widthwisedirection, of the inserted text. In instances in which text is insertedby means of alphanumeric characters input via keypad, the centralprocessing unit 320 determines the number of characters being insertedand the font of the characters and moves the bounding boxes 902containing the handwritten input to the right of the insert further tothe right by an amount equal to the multiple of the number of charactersbeing inserted and the font size of the inserted characters (hereinafterreferenced as the width of the insert). In order to move the boundingboxes 902 containing the handwritten input to the right of the insert tothe right, the x coordinate value of each coordinate pair 900 within abounding box 902 on the same line as and to the right of the insert isincreased by an amount equal to the width of the insert.

Similarly, a handwritten insertion is inserted by initially formingbounding boxes 902 about the strokes 904 that form the handwritten input899. Thereafter, the bounding boxes 902 around existing handwritteninput 899 on the same line as and to the right of the insert are movedfurther to the right by an amount equal to the width of the insert,which is defined in this instance as the difference in the smallest xvalue 900 a associated with the leftmost bounding box 902 of thehandwritten insertion and the largest x value 900 b associated with therightmost bounding box 902 of the handwritten insertion. As describedabove, the bounding boxes 902 of the existing handwritten input 899 thatare being moved are altered by increasing the x coordinate of eachcoordinate pair 900 falling within the bounding boxes 902 by an amountequal to the width of the insert.

In addition to inserting text and/or a handwritten insertion, at least aportion of the existing handwritten input 899 can be deleted and theremaining handwritten input 899 moved so as to close up any gaps in theresulting handwritten input 899. In this regard, the central processingunit 320 effectively moves the bounding boxes 902 surrounding theremaining handwritten input 899 to the left by an amount equal to thewidth of the deleted bounding boxes 902, typically by reducing the xvalues of the coordinate pairs 900 that fall within each of the boundingboxes 902 of the remaining handwriting input 899. In either instance inwhich the editing involves the insertion or deletion of text orhandwritten input 899, the central processing unit 320 can also wrap theresulting handwritten input 899 as appropriate depending upon thepredefined margins and the like. While the above-described editingcapabilities of the present invention relate to the editing of ahandwritten input 899, a textual message that has been previouslyentered can also be edited, such as by inserting handwritten charactersin the same fashion, if so desired.

By providing editing capabilities, the pen-enabled computing device 100and associated methods of the present invention provide increasedfunctionality by permitting a seamless editing of the textual orhandwritten input 899 with either additional text or handwriting andappropriately spacing the resulting data without requiring extensiveinput from the user. Thus, the editing function provided by the penenabled computing device and associated method of the present inventionwill be quite intuitive to a user and will produce a resulting messagethat is appropriately spaced notwithstanding the insertion and deletionof handwriting and/or textual data.

The embodiments described herein address instances in which a singlecomputer program product cooperates with the pen enabled computingdevice to accomplish the functions and processes further describedherein. In some instances, the computer program product may operate,communicate, or otherwise interact with other computer program productssuch as, for example, personal information management (PIM) software orWeb-based PIM or organizer software. These separate computer programproducts may be executing on the pen enabled computing device or on anexternal computing device in communication with the pen enabledcomputing device. As such, in these instances, a myriad of possibleembodiments exist for editing, transferring, or otherwise manipulatingdata related to the nature of a page as identified by the system,computer program products, pen enabled computing device, and associatedmethods as described herein. Thus, it is understood that manipulation ofdata with a plurality of computer program products executing within asingle or plurality of computing devices may provide further associatedembodiments of the present invention as outlined below which will remainconsistent with the spirit and scope of the present invention. Examplesof embodiments for such a system are as follows:

1. Electronic ink based personal information management characterized bycombinations of any or all of the below items:

electronic ink representations of items (e.g., schedule, to-do, contact)are stored in memory on the device for later transmission to personalinformation management (PIM) software running on a personal computer orto a web based organizer.

electronic ink items may be defined by placing forms on the device andspatially segmenting the handwriting which is produced on them (dependson page identification).

2. PIM system for displaying, modifying, augmenting, exporting,synchronizing, insuring consistency with paper items, conditionallytranslating handwriting to text, and printing mixed electronic ink andtext items using PC PIM software or Web based PIM software.

electronic ink items as defined are displayed and can be modified bypersonal information management software on the personal computer or webbased organizer. (items may be defined by segmentation depending on pageid.)

items may be changed using personal information management software onthe personal computer or Web organizer by standard techniques (e.g.,keyboard and mouse editing, speech recognition, etc.

fields of the personal information management software on the personalcomputer or Web organizer accept and display both electronic ink andtext

when an item is changed on the personal computer or Web organizer, thepersonal information management software displays a message whichrecommends that the user print a page containing the item for hisplanner, or change the item on the device using the technique claimedbelow, in order to synchronize the handwritten version of the item withthe electronic version

items may be imported and/or exported to and/or from traditionalpersonal information management software packages

when an electronic ink item is to be exported to another package ordatabase, an automatic on-line handwriting recognizer may be used toconvert the item to a representation more suitable to the externalpackage

user configurable options may be set so that importing and/or exportingto and/or from other packages is automatic

the user may be notified and required to resolve conflicts which mayoccur as a result of importing and/or exporting to and/or from otherpackages

for schedule/calendar items, a message which indicates that a particulartime slot has been reserved (e.g., “reserved”) via handwriting may beexported to an external package. Such a message is sufficient in caseswhere the user wishes to keep the details of his appointments privatefrom other external package users

the personal information management software supports the printing ofelectronic ink based items, which may correspond to different views ofinformation in the database (e.g., items which correspond to a certainperiod of time)

the user may be notified and required to resolve conflicts which mayoccur as a result of synchronizing the database on the PECS with the PCor Web PIM database

the PC PIM or Web Organizer software may or may not organize and displayinformation in the same way as the paper and PECS software. For example,there may be a page labeled “A1” which is the first page of contactswhose last name begins with the letter “A.” The PC PIM software maygroup all contacts whose last name begins with “A” together for displaypurposes. If a change is made on the PC PIM side to handwritteninformation, the PIM software can warn the user that a change toinformation written on page “A1” occurred. Similarly, the user can printinformation from multiple physical pages (e.g., print all of thecontacts whose last name begins with “A”). This implies that physicalpage id is part of the record for a contact.

use keyword button on PECS if the user wants to access database by otherfields—e.g., company name. Could configure software to always tryautomatically recognizing certain fields.

a page printed from the PC PIM software has its identifiers highlightedby some means, so that the user knows which fields to activate in orderto identify the page to the firmware. The PC PIM software may include ameans for assigning a page identifier to a page printed from it. Theprinted page may have as its only electronic page identifier the fieldscorresponding to its identity. Electronic page identifiers can consistof cross-hatched boxes. Location of box can be known by table lookup,using record value (if page id is part of item record) or other keyvalue as index.

3. A system for making changes to electronic ink items on the PECS

changes may be made to items (e.g., schedule, to-do, contact) by tappinga special button on the device (e.g., a cancel button) eitherimmediately preceding or immediately following the indication of theitem (e.g., by drawing one continuous line through the item) “item” ismeaningless without page id—segmentation into items depends on page id

when an item is changed on the handwriting capture device, the PECSdisplay indicates which item was changed

4. A system for making changes to or augmenting machine usable items onthe PECS and reflecting those changes or augmentations in the PC PIM orWeb organizer

when a change should be made to a machine usable item on the device, itcan be deleted and written on paper. PC PIM or Web organizer softwarecan associate the deletion event with the production of new handwritteninformation and attempt to automatically recognize the new piece ofinformation and replace the old information in the PC database with thenew information. “New handwritten information” is associated with aparticular page. Context for automatic recognition will depend on pageid. Alternatively, the user can be asked to type (using the PC or Webaccess device) the new information while viewing an image of thehandwritten information. In either case, the user can be asked toconfirm that the new information should indeed replace the oldinformation in the PC PIM or Web organizer database.

alternatively, the change can be made with buttons on the device orcontrol mechanisms on the paper, for instance a keypad on a preprintedform

if text (previously entered, machine usable) exists in a slot, and theuser ignores the audible warning and message on the display when hewrites in the slot, once the handwriting is uploaded, it may bedisplayed by the PIM or Web Organizer application in a balloon (bubble,tool-tip) near the slot when the cursor is placed near the slot

if text (previously entered, machine usable) exists in a slot, and theuser ignores the audible warning and message on the display when hewrites in the slot, once the handwriting is uploaded, it may bedisplayed by the PIM or Web Organizer application in a nearby field(e.g., a field in a column labeled “additional notes”)

if text (previously entered, machine usable) exists in a slot, and theuser writes in the slot, he may be prompted after an audible warning asto whether or not he wants the handwriting to replace the machine usabletext in the slot. By hitting the “No” button, the user may mean that thehandwriting should be ignored altogether, i.e., the user made a mistakeby writing in said slot.

regarding modifications to a schedule page, because the device has thetime and date available, handwriting in a slot before the time of anappointment can be treated as modifications to the appointment itself,whereas handwriting in a slot during or after the time of theappointment can be treated as notes about the appointment. Modificationsto the appointment are reflected on the device and in the PC PIM or WebOrganizer software after upload, and notes about the appointment aretreated as specified above.

pages printed from the PC PIM software may have holes on the right(equivalently, the printing is on the back side), so that they may beplaced in a ringed binder alongside corresponding pages. For example, ifthe appointment schedule for a user is printed and placed alongside aschedule page for the same day, the user may make notes about theappointments on the right-hand schedule page which are captured by thePECS and treated accordingly by the PC PIM or Web organizer software,while viewing the details of the appointments on the facing page.

5. A system for retrieving and displaying items on the PECS in responseto user input

electronic versions of items may be displayed if available on the PECS,when the user places the pen above or on the item or in response toother activations of control mechanisms. Other information associatedwith the item may optionally be displayed (e.g., date assigned in thecase of a to-do item)

6. A system for reflecting change to item in PC PIM or Web organizer onPECS and on paper

when an item is changed on the personal computer, and the user changesthe item on the device using the technique claimed above, the user mayprefer to make a note or check a box or press a button indicating thatan electronic version of the information is available for display on thedevice.

when changing an item after tapping the cancel button as describedabove, the stroke that identifies the item to be changed may itself bethe note that indicates that an electronic version of the item isavailable.

7. A system for changing the state of certain variables by usingcheckboxes associated with items

a checkbox may be associated with certain items. Its state may betoggled when the pen is placed down in it (e.g., alarm on, alarm off;recurring to-do item on, recurring to-do item off).

the state of a checkbox may be displayed when the pen is placed over orin it (e.g., alarm is on)

8. A system for managing electronic ink based to-do lists characterizedby combinations of any or all of the below items:

placing forms on the device which represent to-do lists with electronicform identifiers and visual form identifiers

segmenting items in the to-do list based on their spatial location onthe form

9. A system for interacting with to-do list items on the PECS

holding the pen over or on some part of a to-do list item givesinformation about the to-do list item, e.g. pen on the item shows thedate the item was assigned on the LCD

marking an item as completed both electronically and on paper bychecking a box associated with the item

marking an item as recurring both electronically and on paper bychecking a box associated with the item

assigning a priority level to an item both electronically and on paperby checking a box associated with the item (e.g., numerical or high,medium, low)

displaying on the LCD the identity of the to-do list form in use (e.g.,“To-do list Nov. 29, 1998”). The identity may be defined by the date andtype of form and indicated by the method claimed above

on a to-do list, if it is assumed that the user writes items in sequenceand does not skip an item or items, then the condition where the userhas failed to correctly identify the page on which he is writing can bedetected if the user writes in a slot which is greater than the nextexpected slot for the current software page. If this condition occurs,the user can be warned appropriately.

10. A system for interacting with to-do list items using PC PIM or Weborganizer software

personal information management software on the personal computer or weborganizer software which allows combined ink and text to-do list itemsto be sorted and viewed by status (complete or incomplete)

personal information management software on the personal computer or weborganizer software which allows combined ink and text to-do list itemsto be sorted and viewed by priority (e.g., high, medium, low)

personal information management software on the personal computer or weborganizer software which allows combined ink and text to-do list itemsto be sorted and viewed by date assigned (Background: the handwritingcapture device associates a date and time with every stroke. The dateassigned can be defined as the date a to-do list item was entered on thelist.)

personal information management software on the personal computer or weborganizer software which allows combined ink and text to-do list itemsto be sorted and viewed by date scheduled to be completed (The datescheduled to be completed can be defined as the date associated with aparticular instance of the to-do list form.)

personal information management software on the personal computer or weborganizer software which allows combined ink and text to-do list itemsto be sorted and viewed by date completed (The date completed can bedefined as the date the completed checkbox was checked, since the deviceassociates a date and time with every stroke.)

personal information management software on the personal computer or weborganizer software which allows a table or tables of combined ink andtext to-do list items to be displayed along with associated values suchas priority, status, date assigned, date scheduled to be completed anddate completed

personal information management software on the personal computer or weborganizer software which allows subsets of combined ink and text to-dolist items to be displayed, possibly along with associated informationdefined above, based on dates (e.g., assigned, scheduled, or completed),i.e. daily, weekly, or monthly views

11. A system for managing electronic ink based address books or contactlists characterized by combinations of any or all of the below items:

placing forms on the device which represent contact lists (see attachedexample) with visual form identifiers and electronic form identifiers

segmenting items in the contact list based on their spatial location onthe form, after the form has been identified by activation of theelectronic form identifier

12. A system for interacting with contact list items on the PECS

holding the pen over or on some part of a contact list item givesinformation about the contact list item, e.g., pen on part of the itemshows the contact's birthday, which may have been entered using thepersonal information management software on the personal computer or weborganizer software

special boxes may be part of a contact list item, and may be used toindicate when certain events related to the contact occur, just byplacing strokes in the boxes, because the device date and time stampseach stroke (e.g., I reach a certain point in the sales cycle with acustomer, I check a box. I contact the customer, I check a box)

13. A system for interacting with contact list items using the PC PIMSoftware or Web Organizer Software

by using the boxes claimed above, the personal information managementsoftware can display contacts which fall into date driven categories(e.g., here are the contacts you were in touch with least recently, hereare the contacts you were in touch with most recently, here are all ofthe contacts with whom you were at a certain point in the sales cycle bythe given date, etc.)

14. A system for displaying information from various databases byinteracting with the forms based PECS and allowing separate navigationcontext and physical page context

on devices which have more physical buttons than existing pen enabledcomputing devices, for example, buttons that allow the user to identifydates and letters of the alphabet, the user may rapidly access machineusable information by tapping those buttons, rather than firstindicating page identifiers on the paper to access machine usableinformation. For example, by tapping the letter “A” in nearly anycontext, the device may display contacts whose last name begins with theletter “A,” and allow the user to scroll through them.

on the device of the previous paragraph, it may be useful to have abutton whose meaning is “return to the previous page identifier.” Forexample, if a user is scheduling an appointment, wants to look up a namein the contacts database by tapping a button, then return to theappointment scheduling task on the current physical page, he returns bytapping said button. Alternatively, the physical paper context may notchange as a result of changing the navigation context, that is, allhandwriting is associated with the physical paper context.

on such devices, it may be useful to always identify pages by writing onthe paper, and use the extra buttons for navigation through the PIMdatabase. In this case, the “OK” button could mean “return to (e.g.,display) the context where the user was writing.” Alternatively, thecurrent paper context could be continuously displayed along with thecurrent “navigation” context, when the two are different. The default isfor the “paper context” and “navigation” contexts to be the same and beindicated by the page change operation.

when the user is using a particular page, and the page context is thesame as the navigation context, the page forward and page backwardbuttons of the PECS allow navigation through the machine usable itemscorresponding to that page or page type, if the current page is not anotes page. The page forward and page backward buttons also allownavigation through the items corresponding to the navigation context, ifit is different from the page context.

Aids for navigation through machine usable data of the typecorresponding to a particular page can be printed on the paper. Forexample, a page of contacts might include a “previous” box, a “next”box, a “name” box, a “phone” box, a “fax” box, and an “e-mail” box.Tapping the “next” box with the pen shows the next contact in machineusable form corresponding to the current page in alphabetical order.Tapping the “fax” and “e-mail” boxes would display the fax number ande-mail address for that contact.

15. A system for providing games for user familiarization, training,entertainment and education: Operation and State Identification

In order to prepare and train the user to make marks in boxes on thepaper to identify pages and navigate through machine usable data, gameswhich require the user to touch certain parts of the paper after aprompt can be included. A game modeled after “Operation” is one example.After the “Operation” page is identified, the display shows the name ofa bone. The user must place the pen in the box corresponding to the bonewithin a certain amount of time or a negative, low frequency sound isheard. Another example is a state identification game. A map of theUnited States is printed on the page. After the map page is identified,the name of a state is displayed and the state must be touched on themap within a certain period of time or a negative, low frequency soundis heard. There are obvious variations of these types of games. Forexample, a state capital name can be displayed and the correspondingstate must be touched.

16. Trip meter application

Another personal information management application is a trip meter. Astarting point and a destination could be touched on a map, and thedriving distance, driving time, and/or flying time between the twopoints could be displayed.

17. Expense tracking

Expenses can be tracked using the PIM system. As long as the user writesdown his expenses on the day they occur, they will be organizedcorrectly by date for the expense reporting software automatically,because the device date and time stamps the writing.

18. A PIM system where user interface elements for controlling the stateof the system are printed on the paper.

Thus, embodiments of the present invention provide a pen enabledcomputing device having the innate capability of processing or otherwisemanipulating recorded pen input data. Embodiments of the presentinvention also provide a pen enabled computing device having thecapability of detecting the presence of a page engaged with the deviceand determining the type of form with a minimum of separateidentification devices and required user-initiated operations. Accordingto a particularly advantageous aspect, the user uses the form accordingto its identified function and does not otherwise have to independentlyindicate the nature of the form to the pen enabled computing device.Accordingly, embodiments of the present invention provide a pen enabledcomputing device having the capability of determining the type of formengaged therewith without requiring additional components or specialmodification of the form to indicate the type of form to the device.Embodiments of the present invention also provide an associated methodof identifying a form to a pen enabled computing device, whileindicating the type of form as well as uniquely identifying forms havingmultiple pages and multiple pages of a single type of form.Advantageously, such a method accomplishes the identification of thepage to the pen enabled computing system while occupying minimal, ifany, space on the writing surface of the page. Thus, embodiments of thepresent invention provide a pen enabled computing device capable ofspecifically identifying the page being used therewith in order tofacilitate future data modification, access, and manipulation in anefficient and expedient manner to provide enhanced seamless andautomatic operation with minimal inconvenience and enhanced flexibilityfor the user.

Many modifications and other embodiments of the invention will come tomind to one skilled in the art to which this invention pertains havingthe benefit of the teachings presented in the foregoing descriptions andthe associated drawings. Therefore, it is to be understood that theinvention is not to be limited to the specific embodiments disclosed andthat modifications and other embodiments are intended to be includedwithin the scope of the appended claims. Although specific terms areemployed herein, they are used in a generic and descriptive sense onlyand not for purposes of limitation.

1. A method for capturing handwritten data, the method comprising:detecting, at a computing device comprising a handwriting captureinterface, a first position associated with an actuation of thehandwriting capture interface to define a handwritten input; in responseto detecting the first position associated with the actuation, definingat least one bounding box encompassing at least a portion of individualstrokes of the handwritten input, wherein a size of the at least onebounding box is based at least on the first position associated with theactuation; and storing the handwritten input and the at least onebounding box.
 2. The method of claim 1, wherein detecting the firstposition associated with the actuation comprises sampling the firstposition associated with the actuation to form a coordinaterepresentation of individual strokes of the handwritten input.
 3. Themethod of claim 2, further comprising determining at least one baselinefor individual strokes of the handwritten input.
 4. The method of claim3, wherein the determining at least one baseline comprises at least oneof detecting a second position associated with the actuation to locatethe at least one baseline or receiving an output from a keypad.
 5. Themethod of claim 3, wherein the determining at least one baselinecomprises detecting changes in a one-dimensional histogram comprising anaxis having a number of predetermined points along the axis, wherein thehistogram comprises a count for individual predetermined points alongthe axis, the count defining the individual strokes of the handwritteninput that pass through the individual predetermined points along theaxis, wherein the at least one baseline is determined relative to apoint along the axis corresponding to at least one predetermined pointhaving a large count relative to other predetermined points.
 6. Themethod of claim 3, wherein the at least one baseline comprises at leasta portion of the at least one bounding box.
 7. The method of claim 3,further comprising modifying the size of the at least one bounding box,wherein at least a portion of the at least one bounding box comprises atleast a portion of the at least one baseline.
 8. The method of claim 1,wherein the computing device comprises a pen-enabled computing device.9. A computing device apparatus comprising: a handwriting captureinterface; a memory element configured to store information associatedwith signals received from the capture interface; and a processingdevice configured to be in communication with the memory element,wherein the processing device is configured to: detect a first positionassociated with an actuation of the handwriting capture interface toform a handwritten input; in response to detection of the first positionassociated with the actuation, define at least one bounding boxencompassing at least a portion of individual strokes of the handwritteninput, wherein a size of the at least one bounding box is based at leaston the first position associated with the actuation; and store thehandwritten input and the at least one bounding box in the memoryelement.
 10. The apparatus of claim 9, wherein the processing device isfurther configured to sample the first position associated with theactuation of the handwriting capture interface to form a coordinaterepresentation of individual strokes of the handwritten input.
 11. Theapparatus of claim 10, wherein the processing device is furtherconfigured to determine at least one baseline for individual strokes ofthe handwritten input, and the at least one baseline comprises alocation and at least a portion of the at least one bounding box. 12.The apparatus of claim 11, wherein the processing device is furtherconfigured to at least one of detect a second position associated withthe actuation to define the location of the at least one baseline orreceive an output from a keypad to define the location of the at leastone baseline.
 13. The apparatus of claim 11, wherein the location of theat least one baseline is based on changes in a one-dimensional histogramcomprising an axis having a number of predetermined points along theaxis, wherein the histogram comprises a count for individualpredetermined points along the axis, the count defining the individualstrokes of the handwritten input that pass through the individualpredetermined points along the axis, wherein the location of the atleast one baseline is determined relative to a point along the axiscorresponding to the predetermined point having a large count relativeto other predetermined points.
 14. The system of claim 11, wherein theat least one baseline comprises at least a portion of the at least onebounding box.
 15. The system of claim 11, further comprising modifyingthe size of the at least one bounding box, wherein at least a portion ofthe bounding box comprises at least a portion of the at least onebaseline.
 16. A tangible computer readable medium configured to storeinstructions configured to cause a computing device to performoperations comprising: detecting a first position associated with anactuation of a handwriting capture interface in communication with thecomputing device to define a handwritten input comprising a coordinaterepresentation of individual strokes of the handwritten input; inresponse to detecting the first position associated with the actuation,defining at least one bounding box encompassing at least a portion ofindividual strokes of the handwritten input, wherein a size of the atleast one bounding box is based at least on the first positionassociated with the actuation; and storing the handwritten input and theat least one bounding box.
 17. The computer readable medium of claim 16,further configured to cause the computing device to perform operationscomprising determining a location of at least one baseline based on thecoordinate representation of individual strokes of the handwritteninput.
 18. The computer readable medium of claim 17, wherein thedetermining the location comprises at least one of detecting a secondposition of the actuation to define the location or receiving an outputfrom a keypad to define to the location.
 19. The method of claim 17,wherein the determining the location comprises detecting changes in aone-dimensional histogram comprising an axis having a number ofpredetermined points along the axis, wherein the histogram comprises acount for individual predetermined points along the axis, the countdefining the individual strokes of the handwritten input that passthrough the individual predetermined points along the axis, wherein theat least one baseline is determined relative to a point along the axiscorresponding to at least one predetermined point having a large countrelative to other predetermined points.